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ABSTRACT 

The  problem  of  calibrating  two  color  cameras  as  a  stereo  pair  has  been  heavily  researched  and  many  off-the-shelf 
software  packages,  such  as  Robot  Operating  System  and  OpenCV,  include  calibration  routines  that  work  in 
most  cases.  However,  the  problem  of  calibrating  two  infrared  (IR)  cameras  for  the  purposes  of  sensor  fusion  and 
point  could  generation  is  relatively  new  and  many  challenges  exist.  We  present  a  comparison  of  color  camera 
and  IR  camera  stereo  calibration  using  data  from  an  unmanned  ground  vehicle.  There  are  two  main  challenges 
in  IR  stereo  calibration;  the  calibration  board  (material,  design,  etc.)  and  the  accuracy  of  calibration  pattern 
detection.  We  present  our  analysis  of  these  challenges  along  with  our  IR  stereo  calibration  methodology.  Finally, 
we  present  our  results  both  visually  and  analytically  with  computed  reprojection  errors. 
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1.  INTRODUCTION 

The  area  of  autonomous  navigation  for  unmanned  ground  vehicles  (UGV)  is  a  rapidly  growing  field  of  research. 
There  are  many  successful  programs,  such  as  the  DARPA  Grand  Ghallenge^  and  the  Google  driverless  car,^ 
that  are  helping  to  drive  this  research  into  a  reality.  The  US  Navy,  Department  of  Defense  (DoD),  and  the 
Department  of  Homeland  Security  (DHS)  are  also  becoming  increasingly  interested  in  unmanned  vehicles  for 
various  applications. 

Many  challenges  still  persist  in  the  area  of  autonomous  (and  even  semi-autonomous)  vehicle  navigation  for 
UGVs.  One  challenge  is  in  detecting  and  classifying  obstacles  for  avoidance  and  path  planning.  The  use  of 
laser-based  sensors,  such  as  Lidar,  has  become  quite  common  for  assisting  in  such  a  task,  however,  Lidar  systems 
may  be  too  expensive  for  certain  application  and  are  active,  not  passive,  sensors,  so  they  may  not  be  desirable 
in  some  missions.  Lidars  are  adversely  effected  by  smoke,  dust,  fog,  and  rain.  Therefore,  the  use  of  passive 
camera  sensors,  such  as  typical  color  and  infrared  (IR)  cameras,  has  become  an  important  research  topic  in 
UGV  navigation.  However,  to  build  three-dimensional  (3D)  representations  of  the  scene,  a  pair  of  stereo  cameras 
(color,  IR,  or  otherwise)  must  be  accurately  calibrated  with  respect  to  the  rotation  and  translation  between 
the  two  sensors.  These  are  also  known  as  the  extrinsic  parameters,  and  once  known  a  disparity  map  may  be 
computed  from  two  synchronized  images  from  the  cameras.  Once  a  disparity  map  is  known,  a  depth  image  (3D 
points)  may  be  computed. 

One  of  the  greatest  challenges  of  using  a  stereo  pair  of  color  and/or  IR  cameras  is  to  accurately  determine 
the  extrinsic  calibration  parameters  between  the  pair  of  cameras.  For  color  cameras,  this  has  historically  been 
solved  using  a  checkerboard  pattern  of  black  and  white  squares  using  packages  such  as  the  Gamera  Galibration 
Toolbox  for  Matlab®,^  OpenGV,^  and  the  Robot  Operating  System  (ROS).^  However,  the  same  packages  do 
not  necessarily  work  out-of-the-box  for  IR  stereo  cameras,  due  to  thermal  radiation  required  for  high  and  low 
intensity  pixels  in  an  IR  sensor.  For  instance,  on  a  cold  and  cloudy  day,  there  will  be  very  little  difference 
registered  in  an  IR  sensor  between  the  black  and  white  squares  on  a  piece  of  paper.  Therefore,  more  care  and 
preparation  is  required  in  order  to  calibrate  stereo  IR  cameras. 
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This  paper  is  organized  in  the  following  manner.  First,  we  briefly  introduce  related  works  in  the  area  of  IR 
stereo  camera  calibration  in  Section  2.  In  Section  4,  we  discuss  the  two  main  challenges  encountered  in  IR  stereo 
calibration  and  our  final  methodology  for  IR  stereo  calibration.  Finally,  we  present  our  results  in  Section  5  and 
conclude  the  paper  in  Section  6. 


2.  RELATED  WORK 

As  previously  noted,  research  into  calibrating  the  extrinsic  parameters  for  color  stereo  cameras  is  well  estab¬ 
lished  and  vast.  Existing  calibration  packages  and  libraries  found  in  ROS  and  OpenCV  have  become  standard 
approaches  for  computing  extrinsic  parameters  in  many  practical  applications.  However,  IR  stereo  calibration  is 
a  fairly  new  topic  and  is  sparsely  represented  in  literature. 

One  of  the  most  extensive  works  on  the  subject  is  from  Rankin  et  al.,^  where  the  authors  the  details  and 
applications  of  their  perception  system  which  relies  on  thermal  and  IR  cameras  (which  they  refer  to  as  TIR). 
Their  first  TIR  stereo  calibration  target  consisted  of  a  board  with  a  pattern  of  symmetric  circles  with  a  light 
bulb  placed  behind  each  plastic  circle  insert.  Another  pattern  simply  consisted  of  pattern  of  symmetric  circles, 
much  like  one  of  the  patterns  we  use  in  our  experiments.  Their  final  calibration  target  used  a  checkerboard 
pattern  background  with  a  grid  of  fine  wires  around  the  boarders  of  the  checker  squares  which,  when  current 
is  applied,  provides  a  very  clear  calibration  target  for  both  color  and  IR  cameras.  Ursine  et  al.^  introduce 
a  methodology  for  reliably  calibrating  color  and  IR  stereo  cameras  using  a  checker  calibration  board  with  a 
copper  plate  background  (with  low  emissivity)  and  black  squares  painted  on  with  a  high  emissivity  spray  ink. 
This  way,  both  the  color  and  IR  cameras  may  view  the  pattern  and  attain  a  similar  differential  between  the 
boundaries  of  the  squares  for  pattern  detection.  Another  example  IR  stereo  system,  developed  by  Kong  et  al.,^ 
is  a  system  for  autonomous  landing  of  an  unmanned  aerial  vehicle  using  a  ground-based  IR  stereo  system.  Their 
methodology  for  calibrating  the  extrinsic  parameters  for  the  IR  cameras  was  to  paste  black  squares  on  a  mirror 
and  then  actively  heat  the  mirror  to  achieve  the  classic  checker  board  pattern  commonly  used  in  calibration. 
The  accuracy  of  their  calibration  was  not  discussed  in  the  paper.  In  the  work  of  Starr  and  Lattimer,^  a  pair  of 
IR  cameras  are  used  to  compare  visibility  in  fire  conditions  between  IR  stereo  and  Lidar  sensor  systems.  They 
calibrate  the  cameras  using  a  pattern  that  is  formed  from  a  cold  metal  grid  being  placed  in  front  of  a  black  body 
radiation  source  (liquid-crystal  display  computer  monitor).  Another  application  of  IR  stereo  cameras  is  found 
in  the  work  of  Bertozzi,  Broggi,  and  Lasagni.^^  In  their  work,  the  extrinsic  parameters  between  two  IR  cameras 
are  used  to  triangulate  the  position  and  distance  of  detected  objects  in  a  scene  which  further  assists  in  refining 
their  methodology  for  pedestrian  detection  from  their  IR  camera  system.  While  their  results  look  promising, 
their  methodology  for  calibrating  the  two  IR  stereo  cameras  is  not  discussed  in  the  paper. 

In  our  work,  we  had  limited  resources  available  for  heating,  illuminating  and  otherwise  building  a  calibration 
board  from  new  materials  from  scratch.  Instead,  we  present  an  analysis  using  three  black  and  white  calibration 
patterns;  checkerboard,  symmetric  circles,  and  asymmetric  circles. 

3.  SYSTEM  DESCRIPTION 

Our  UGV  is  based  on  the  TORC  By  Wire  XGV^^,  shown  in  Figure  1(a).  For  IR  imagery,  we  use  two  FLIR® 
Tan®  640  35mm  cameras  with  a  baseline  of  approximately  0.5m.  For  color  imagery,  we  use  two  Prosilica  GG750 
cameras  from  Allied  Vision  Technologies  with  a  baseline  of  approximately  Im.  There  are  several  challenges 
inherent  to  such  a  system,  such  as  the  wide  baselines  and  outdoor  weather  conditions,  that  will  be  addressed  in 
this  paper.  Synchronization  of  the  cameras  is  assumed  for  the  content  of  this  work. 

4.  CHALLENGES  AND  METHODOLOGY 

There  are  numerous  challenges  present  when  attempting  to  calibrate  a  stereo  pair  of  IR  cameras.  In  this  paper, 
we  will  cover  the  two  most  difficult  challenges  found  in  our  experience;  the  calibration  board  (material,  design, 
etc.)  and  the  accuracy  of  calibration  pattern  detection. 


(a)  (b) 

Figure  1.  TORC  By  Wire  XGV  (a)  and  camera  sensors  (b) 


4.1  Calibration  Board 

The  first  challenge,  that  has  been  mentioned  by  several  authors  previously,  is  the  calibration  board  itself.  Unlike 
the  calibration  pattern  for  color  stereo  cameras,  which  can  utilize  simple  black  and  white  checkerboard  patterns 
for  highly  accurate  calibration,  the  calibration  pattern  for  two  IR  stereo  cameras  must  be  carefully  selected, 
designed,  and/or  manufactured.  Rankin  et  al.^  use  small  light  bulbs  behind  a  circular  pattern  and  a  grid  of 
wires  placed  on  a  checkerboard  pattern.  Ursine  et  al.^  use  a  checkerboard  pattern  painted  on  a  copper  plate 
background.  Kong  et  al.^  use  a  heated  mirror  with  black  squares  painted  on  for  a  checkerboard  pattern.  And 
finally  Starr  and  Lattimer^  use  a  cold  metal  grid  heated  by  a  computer  monitor.  Related  to  the  above  challenges 
are  the  wide  baseline  between  the  two  cameras,  which  requires  larger  calibration  boards  than  standard  boards, 
and  the  many  challenges  faced  with  calibrating  in  an  outdoor  environment,  such  as  uncontrolled  lighting,  wind, 
dust,  and  other  difficult  to  predict  weather  conditions.  While  we  did  not  have  the  resources  to  build  the  patterns 
described  by  the  authors  above,  we  were  able  find  a  solution  to  these  challenges,  which  we  will  describe  below. 

4.2  Calibration  Pattern 

The  second  challenge  we  encountered  was  the  calibration  pattern  itself.  We  started  with  the  classic  black  and 
white  checkerboard  pattern,  but  the  dynamic  range  between  the  “white”  and  “black”  squares  was  not  sufficient 
for  the  calibration  routine  to  detect  the  pattern.  Even  when  we  increased  the  size  of  the  calibration  board 
from  10cm  squares  to  15cm  squares,  the  detection  algorithm  was  unsuccessful.  An  example  of  this  challenge  in 
dynamic  range  is  shown  in  Figure  6.  Next  we  tried  a  pattern  of  symmetric  circles,  but  since  each  circle  was  only 
5.08cm  in  diameter,  they  were  difficult  to  detect  at  the  distances  needed  for  the  baseline  between  the  two  cameras. 
Heating  was  tried  (both  with  a  portable  heating  device  and  direct  sunlight)  on  both  the  checkerboard  pattern 
and  the  symmetric  circles  pattern,  but  the  calibration  routine  still  failed  to  detect  the  patterns  successfully.  In 
addition  to  the  trouble  with  our  available  calibration  patterns,  the  software  we  used  to  compute  the  calibration 
suffered  from  setbacks  as  well.  For  instance,  the  “Fuerte”  version  of  ROS  we  used  did  not  support  circle  patterns 
for  calibration,  so  we  used  OpenCV  to  compute  the  calibration  parameters  offline. 

4.3  Final  Methodology 

Our  final  methodology  that  was  successful  in  calibrating  our  IR  stereo  cameras  was  the  following.  First,  we  used 
a  calibration  board  manufactured  by  858  Graphics,  Inc.  that  was  made  from  dibond,  which  is  a  lightweight,  rigid 
and  durable  aluminum  composite  material.  The  board  was  ordered  to  be  \in  white  dibond  with  no  laminate. 
858  Graphics,  Inc.  recently  switched  to  a  new  manufacturing  process  where  they  print  the  patterns  directly  onto 
the  dibond  and  this  resulted  in  a  much  improved  ability  to  detect  the  patterns  with  IR  cameras.  The  pattern 
printed  onto  the  boards  was  a  3  by  5  pattern  of  asymmetric  circles  with  a  17cm  diameter  with  a  spacing  of  17cm 
between  the  circles.  Also,  we  found  that  by  using  this  large  asymmetric  circles  pattern  on  a  warm  day  with  little 


to  no  wind  (with  the  board  left  in  direct  sunlight),  the  results  detection  results  improved  dramatically.  Example 
data  taken  from  two  IR  cameras  of  the  final  calibration  board  in  good  conditions  is  found  in  Figure  7. 

Additionally,  simple  preprocessing  techniques  were  used  to  try  to  increase  the  accuracy  of  IR  stereo  calibra¬ 
tion.  Three  methods  were  used  and  are  briefly  described.  In  the  first  method,  we  applied  a  median  blur  filter  to 
each  IR  image  with  a  window  of  5  pixels.  For  the  second  method,  we  used  a  thresholding  function  to  truncate 
the  pixel  values  above  an  intensity  of  50.  The  third  method  simply  combines  the  first  two  methods.  Example 
images  of  the  median  blur  filter,  thresholding,  and  the  combination  of  the  two  are  shown  in  Figures  2(b),  2(c), 
and  2(d),  respectively. 


(c)  (d) 


Figure  2.  Example  images  of  IR  image  preprocessing;  (a)  original  image,  (b)  median  blurred  image,  (c)  thresholded  image, 
and  (d)  combined 


5.  RESULTS 

For  our  experiments  with  the  color  cameras  in  our  system,  checkerboard  patterns,  symmetric  circle  patterns  and 
asymmetric  circle  patterns  where  all  successfully  used  in  our  calibration  routine.  Example  images  of  each  of 
these  patterns  is  shown  in  Figures  3,  4,  and  5,  respectively.  For  our  experiments  with  the  IR  stereo  cameras, 
only  the  asymmetric  circle  patterns  were  successfully  used  in  calibrating  the  cameras.  Examples  of  these  images 
are  found  in  Figure  7.  An  example  of  a  failure  mode  with  the  checkerboard  pattern  in  the  left  IR  camera  is 
shown  in  Figure  6. 


To  evaluate  the  stereo  calibration  results  numerically,  we  used  OpenCV  to  calculate  the  stereo  reprojection 
error  for  each  of  the  calibration  patterns  used  for  both  the  color  stereo  cameras  and  the  IR  stereo  cameras.  The 
OpeCV  function  “stereoCalibrate”  returns  the  RMS  reprojection  error  which  is  computed  by  transforming  the 
3D  points  of  the  calibration  board  using  the  computed  extrinsic  parameters  to  both  the  left  and  right  image  and 
comparing  those  projected  points  to  the  detected  corners  or  circle  centroids. 

The  results  are  shown  in  Tables  1  and  2. 


Calibration  pattern 

Image  pairs  detected  / 
total  images 

Dim  (wxh) 

Pattern  size  (cm) 

RMS  reprojection 
error 

Checkerboard 

38/38 

6x5 

15 

0.810 

Small  circles 

42/42 

10x9 

5.08 

0.458 

Asymmetric  circles 

53/60 

3x5 

17 

0.348 

Table  1.  Stereo  reprojection  error  of  color  stereo  cameras 


Calibration  pattern 

Image  pairs  detected  / 
total  images 

Dim  (wxh) 

Pattern  size  (cm) 

RMS  reprojection 
error 

Checkerboard 

0/38 

6x5 

15 

N/A 

Small  circles 

0/42 

10x9 

5.08 

N/A 

Asymmetric  circles 

40/129 

3x5 

17 

3.683 

Asymmetric  circles 
(median  blur) 

21/129 

3x5 

17 

0.743 

Asymmetric  circles 
(threshold) 

40/129 

3x5 

17 

0.501 

Asymmetric  circles 
(median  blur+threshold) 

68/129 

3x5 

17 

0.239 

Table  2.  Stereo  reprojection  error  of  IR  stereo  cameras 


In  Table  2,  the  reprojection  errors  were  “N/A”  for  the  calibration  patterns  which  were  not  detected  by  the 
calibration  routine.  However,  the  asymmetric  circle  patterns  were  successfully  detected  with  and  without  image 
preprocessing.  Clearly,  the  simple  preprocessing  done  to  the  IR  imagery  improved  the  IR  stereo  calibration  re¬ 
sults.  As  shown  in  Table  1,  even  the  color  stereo  cameras  benefit  in  reprojection  error  when  using  the  asymmetric 
board  for  stereo  calibration. 


6.  CONCLUSION  AND  FUTURE  WORK 

We  have  presented  our  analysis  and  methodology  for  calibrating  a  pair  of  IR  stereo  cameras.  The  two  most  diffi¬ 
cult  challenges  faced  in  this  work  were  the  calibration  board  and  the  calibration  pattern.  Our  final  methodology 
uses  an  asymmetric  calibration  pattern  printed  on  a  dibond  board.  When  left  in  direct  sunlight  before  calibra¬ 
tion,  the  board  is  used  to  successfully  calibrate  our  IR  stereo  cameras  with  no  image  preprocessing  resulting  in 
acceptable  stereo  reprojection  error.  For  future  work,  we  have  plans  to  integrate  our  OpenCV  calibration  routine 
into  ROS  and  implement  further  preprocessing  techniques  for  the  IR  imagery,  such  as  adaptive  thresholding  and 
background  subtraction.  Additionally,  we  plan  on  exploring  the  next  steps  of  feature  matching  for  creating 
disparity  maps  which  will  lead  to  the  generation  of  point  clouds  from  our  IR  stereo  cameras.  We  also  plan  on 
exploring  other  easy- to- manufacture  materials  for  calibration  of  both  color  and  IR  stereo  cameras.  Additionally, 
we  would  like  to  explore  other  calibration  patterns  which  might  be  beneficial. 
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Figure  3.  Checker  board  pattern  for  left  (a)  and  right  (b)  color  cameras 


Figure  4.  Symmetric  circle  board  pattern  for  left  (a)  and  right  (b)  color  cameras 


Figure  6.  Example  image  of  the  checker  board  pattern  from  left  IR  camera 
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Figure  7.  Asymmetric  circle  board  pattern  for  left  (a)  and  right  (b)  IR  stereo  cameras 


